Goto

Collaborating Authors

 16-bit precision


Pytorch Training Tricks and Tips

#artificialintelligence

In this article, I will describe and show the code for 4 different Pytorch training tricks that I personally have found to improve the training of my deep learning model. In a regular training loop, PyTorch stores all float variables in 32-bit precision. For people who are training their models with strict constraints, sometimes, this can cause their model to take up too much memory, forcing them to have a slower training process with a smaller model and a smaller batch size. However, storing all the variables/numbers in the model in 16-bit precision can improve upon and fix most of these problems, like dramatically decreasing the memory consumption of the model and speeding up the training loop while still maintaining the same performance/accuracy of the model. Converting all calculations to 16-bit precision in Pytorch is very simple to do and only requires a few lines of code.